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DETAILED ACTION 

1 . Claims 7 and 8 are canceled while the original claims 1 and 2, amended claims 3-6 and 
the added claim 9 are pending for examination. 

Specification 

2. The abstract of the disclosure is objected to because it contains numbered reference 
designations which should not be used in an abstract. Correction is required. See MPEP 

§ 608.01(b). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be oblaiiiccl iliDUgh the cniion is ik)1 idciuically disck)scd or described as set forth in 
section 102 of this title, if the dift'crcnccs between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

3. Claims 1-6 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Jaszlics 
et al. (U. S. Patent 6,166,744, hereafter '744) and fiirther in view of Lescinsky et al. ("Interactive 
Scene Manipulation in the Virtue3D System," ACM Weg3D'02, pp. 127-135, February 24-28, 
2002, Tempe, Arizona, ACM 1-58113-468-1/02/0002, hereafter '468). 
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4. Regarding claim 1 (Original), Jaszlics teaches an image processing method of presenting 
a virtual object superimposed on a physical space to an observer ('744 Abstract, col. 10, In. 1-9), 
characterized by comprising: a holding step of holding, in a memory (global memory, '744; fig. 
20), information of a state of each virtual object included in a virtual space ('744; col. 14, In. 32- 
52); an association step of associating at least one virtual object included in the virtual space with 
at least one position/orientation sensor which is held in a hand of the observer and manipulated 
(joystick; '744; col. 10, In. 35-38); col. 1 1, In. 47-56); a layout step of laying out, in the virtual 
space, the virtual object associated with the position/orientation sensor in the association step in 
accordance with a position and orientation of the position/orientation sensor itself, which are 
determined on the basis of a measurement result by the position/orientation sensor ('744; col. 10, 
In. 35-38; col. 1 1, In. 47-56); but does not teach a presentation step of presenting the state of each 
virtual object included in the virtual space to the observer on the basis of the information held in 
the holding step; an input step of inputting an operation to change a desired one of the states 
presented in the presentation step; and continues to teach an updating step of updating the 
information in accordance with the operation input in the input step ('744; col. 1 1, In. 47-56). 
Lescinsky, working in the same field of endeavor, however, teaches a presentation step of 
presenting the state of each virtual object included in the virtual space to the observer on the 
basis of the information held in the holding step ('468; fig. 4; pg. 313, col. 2, § 8; pg. 132, col. 1, 
§ 9); for the benefit of allowing the user to easily edit and control virtual objects and the 
composed scene in a natural way. It would have been obvious to one of ordinary skill in the art 
at the time of the invention to combine the rich teachings of Jaszlics' virtual and physical scene 
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simulation with the scene manipulation teachings of Lescinsky for the benefit of allowing the 
user to easily edit and control virtual objects and the composed scene in a natural way. 

5. In regards to claim 2 (Original), Jaszlics further teaches the method characterized by 
further comprising a registration step of registering, in the information, the position and 
orientation of the position/orientation sensor itself, which are determined on the basis of the 
measurement resuU by the position/orientation sensor as a position and orientation of the virtual 
object associated with the position/orientation sensor in the association step, wherein in the 
layout step, the virtual object is laid out in the virtual space by using the position and orientation 
of the virtual object registered in the information in the registration step ('744; col. 1 1, In. 30- 
63). 

6. Regarding claim 3 (Currently Amended), Jaszlics and Lescinsky teach the method 
according to claim 1 and Lescinsky further teaches wherein the information includes a name 
(model ID) ('486; pg. 134, col. 1, "To set and get Model Properties"), but does not teach blinking 
state, selected/unselected state, and section display state of the virtual object. It would have been 
obvious to one of ordinary skill in the art at the time of the invention to use the general 
properties, methods, constraints and scene methods structures laid out in Lescinsky ('486; pg. 
134, col. 1 and 2) to define blinking state, selected/unselected state, and section display state of 
the virtual object objects and methods for the benefit of instancing these attributes and allowing 
user to control these attributes. The VRML construct generation module in fig. 40, element 246 
('744; col. 14, In. 40-52) assembles data strings describing the individual virtual objects (VMO), 
handling the VRML constructs taught by Lescinsky). 
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7. In regards to claim 4 (Currently Amended), Jaszlics teaches an image processing method, 
including the steps of: acquiring position and orientation information of an observer ('744; col. 
10, In. 52-60), acquiring an image of a physical space ('744; col. 9, In. 42-47); generating an 
image of a virtual space in accordance with the position and orientation information of the 
observer ('744; col. 9, In. 32-57); compositing the image of the physical space and the image of 
the virtual space and displaying the composited image on a head mounted display worn by a user 
('744; col. 9, In. 32-57); generating an operation panel image and compositing the operation 
panel image with the image of the physical space and the image of the virtual space ('744; fig. 
21; col. 15, In. 5-25); acquiring position information of an operation unit operated by the 
observer('744; col. 1 1, In. 30-63); and updating the operation panel image in accordance with a 
positional relationship between the operation panel image and the operation unit ('744; col. 15, 
In. 3 1-47), but does not teach wherein in the operation panel image, a part selected by the 
operation unit is enlarged. Jaszlics, however, teaches magnification of virtual gun sights under 
user control ('744, table 3, col. 1 1, In. 24-30). It would have been obvious to one of ordinary 
skill in the art at the time of the invention to extend Jaszlics teachings of gun sight magnification 
and provide magnification of an area selected by a user or user motions for the benefit of 
enhancing the simulation process. 

8. Regarding claim 5 (Currently Amended), Jaszlics teaches the method according to claim 
4, wherein the image of the virtual space is generated on the basis of a 3D CAD data of a virtual 
object ('744; claim 19; virtual reality modeling language (VRML), but does not teach the 
operation panel contains an assembly tree based on the 3D CAD data, and the enlarged part 
includes a component name contained in the assembly tree. Lescinsky, however, teaches the 
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operation panel contains an assembly tree based on the 3D CAD data, and the enlarged part 
includes a component name contained in the assembly tree for the benefit of allowing the user to 
easily edit and manipulate the properties of the virtual objects ('468; fig. 4, pg. 133, § 9). 
9. Regarding claim 6 (Currently Amended), Jaszlics teaches an image processing apparatus 
for presenting a virtual object superimposed in a physical space to an observer ('744 Abstract, 
col. 10, In. 1-9), comprising: a holding unit adapted to hold information of a state of each virtual 
object included in a virtual space (global memory, '744; fig. 20); an association unit (virtual 
simulation computer; '744; fig. 16, element 210) adapted to associate at least one virtual object 
included in the virtual space with at least one position/orientation sensor (joystick; '744; col. 1 0, 
In. 35-38); col. 1 1, In. 47-56) manipulated by the observer (virtual simulation computer; '744; 
fig. 16, element 210, implementing the processes of Fig. 20); a layout unit (virtual simulation 
computer; '744; fig. 16, element 210) adapted to lay out, in the virtual space, the virtual object 
associated with the position/orientation sensor by said association unit in accordance with a 
position and orientation of the position/orientation sensor itself, which are determined on the 
basis of a measurement result by the position/orientation sensor (virtual simulation computer; 
'744; fig. 16, element 210, implementing the processes of Fig. 20); a presentation unit (virtual 
simulation computer and display; '744; fig. 16, elements 210 and 211) but does not teach 
adapted to present the state of each virtual object included in the virtual space to the observer on 
the basis of the information held by said holding unit; and continues to teach an input unit 
(joystick; '744; col. 10, In. 35-38); col. 1 1, In. 47-56) adapted to input an operation to change a 
desired one of the states presented by said presentation unit (virtual simulation computer and 
display; '744; fig. 16, elements 210 and 211, implementing the processes of Fig. 20); and an 
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updating unit (virtual simulation computer; '744; fig. 16, element 210) adapted to update the 
information in accordance with the operation input by said input unit (virtual simulation 
computer; '744; fig. 16, element 210, implementing the processes of Fig. 20). Lescinsky, 
working in the same field of endeavor, however, teaches a presentation step of presenting the 
state of each virtual object included in the virtual space to the observer on the basis of the 
information held in the holding step ('468; fig. 4; pg. 313, col. 2, § 8; pg. 132, col. 1, § 9); for the 
benefit of allowing the user to easily edit and control virtual objects and the composed scene in a 
natural way. It would have been obvious to one of ordinary skill in the art at the time of the 
invention to combine the rich teachings of Jaszlics' virtual and physical scene simulation with 
the scene manipulation teachings of Lescinsky for the benefit of allowing the user to easily edit 
and control virtual objects and the composed scene in a natural way. 

10. Claims 7-8. (canceled) 

1 1 . Regarding claim 9 (New), Jaszlics teaches a computer-readable medium encoded with a 
computer program for causing a computer to execute an image processing method of presenting 
a virtual object superimposed on a physical space to an observer ('744 Abstract, col. 10, In. 1-9), 
comprising: a holding step of holding, in a memory (global memory, '744; fig. 20), information 
of a state of each virtual object included in a virtual space ('744; col. 14, In. 32-52); an 
association step of associating at least one virtual object included in the virtual space with at least 
one position/orientation sensor which is held in a hand of the observer and manipulated (joystick; 
'744; col. 10, In. 35-38); col. 1 1, In. 47-56); a layout step of laying out, in the virtual space, the 
virtual object associated with the position/orientation sensor in the association step in accordance 
with a position and orientation of the position/orientation sensor itself, which are determined on 



Application/Control Number: 10/594,1 14 Page 8 

Art Unit: 2628 

the basis of a measurement result by the position/orientation sensor ('744; col. 10, In. 35-38; col. 
1 1, In. 47-56); but does not teach a presentation step of presenting the state of each virtual object 
included in the virtual space to the observer on the basis of the information held in the holding 
step; an input step of inputting an operation to change a desired one of the states presented in the 
presentation step; and continues to teach an updating step of updating the information in 
accordance with the operation input in the input step ('744; col. 1 1, In. 47-56). Lescinsky, 
working in the same field of endeavor, however, teaches a presentation step of presenting the 
state of each virtual object included in the virtual space to the observer on the basis of the 
information held in the holding step ('468; fig. 4; pg. 313, col. 2, § 8; pg. 132, col. 1, § 9); for the 
benefit of allowing the user to easily edit and control virtual objects and the composed scene in a 
natural way. It would have been obvious to one of ordinary skill in the art at the time of the 
invention to combine the rich teachings of Jaszlics' virtual and physical scene simulation with 
the scene manipulation teachings of Lescinsky for the benefit of allowing the user to easily edit 
and control virtual objects and the composed scene in a natural way. 

Conclusion 

The following prior art, made of record, was not relied upon but is considered pertinent to 

applicant's disclosure: 

US 20030142067 Al Three dimensional volumetric display input and output 

configurations ~ Directed to input and output configurations for 
three-dimensional volumetric displays that allow the content of a 
three-dimensional volumetric display to be affected by actions by a 
user. 

US 6466232 Bl Method and system for controlling presentation of information to a 

user based on the user's condition - The user's condition is 
determined by integrating the output of a large number of sensors 
associated with the user. 
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US 6408257 Bl Augmented-reality display method and system 

US 5590062 A Simulator for producing various living environments mainly for 

visual perception - Virtual reality construction of 3D spaces for 
designers and clients allowing the building and manipulation of the 
3D scenes by the user in real-time via a heads-up display and hand 
controller device simulator environment. 

Althoff et al.; "A Generic Approach for Interfacing VRML Browsers to Various Input 
Devices and Creating Customizable 3D Applications," ACM 
Weg3D'02, pp. 67-74, February 24-28, 2002, Tempe, Arizona, 
ACM 1-58113-468-1/02/0002 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Edward Martello whose telephone number is (571) 270-1883. 
The examiner can normally be reached on M-F 7:30-5:00 EST. 

If attempts to reach the examiner by telephone are unsuccessfiil, the examiner's 
supervisor, Xiao Wu can be reached on (571) 272-7761 . The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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